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(54) Multimedia conferencing using parallel networks 



(57) A conference server system, for internal use in 
the public switched telephone network (PSTN), links to 
public data communication networks (e.g. the Internet) 
for distributing computer displayable data between par- 
ticipants in voice telephone conferences. The server 
system includes a computer sub-system for storing data 
generated by the participants and distributing the data 
in coordination with voice presentations of respective 
participants. Data so stored and distributed includes 
computer-displayable data. Other data handled by the 
computer subsystem represents commands and func- 
tional requests issued by participants which pertain to 
the handling of computer-displayable data and voice pa- 
rameters of the conference, conference participants, 
having separate and concurrent access to both the 
PSTN and the data network, receive and view compu- 
ter-displayable data prepared by one of the participants 
in coordination with the respective voice conference. 
The conference server system connects to voice con- 
ference management resources of the PSTN in a man- 
ner enabling the server system to control voice param- 
eters of a voice conference in response to data requests 
received from participants (e.g. requests from hearing- 
impaired participants for enhanced audio service), as 
well as enabling the conference management resources 
to control functions performed in respect to distribution 
of displayable data in response to spoken requests ut- 
tered by the participants, when the telephone system 
contains appropriate speech recognition equipment. 



Figure 3 
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agement facilities for managing operations in said con- 
ferencing circuits to provide conference connections be- 
tween end users requiring said connections; said con- 
ference server system comprising: a computer system 
for storing and processing data; means connecting said 
computer system to a data network external to said 
PSTN for transferring data between said computer sys- 
tem and end users of said PSTN currently participating 
in a voice conference utilizing said conferencing circuits; 
said data being transferred in parallel over said data net- 
work while signals representing voice utterances of said 
participating end users are transmitted through said 
conferencing circuits; said data including displayabie 
image data and data representing functional requests 
and commands issued by individual said participating 
end users and requiring attention by said conference 
server system; said functional requests including re- 
quests from a said participating end user to said confer- 
ence server system to have said conference server sys- 
tem distribute displayabie image data, previously fur- 
nished to said conference server system by the partici- 
pating end user issuing the request, to others of said 
participating end users; and means linking said compu- 
ter system to said conferencing circuits and said confer- 
ence management facilities for enabling said computer 
system to coordinate distribution of said displayabie im- 
age data between said participating end users in coor- 
dination with voice interchanges between said partici- 
pating end users; said linking means also being useful 
for enabling said computer system to participate in sat- 
isfaction of functional requests and commands issued 
by said participating end users and said conference 
management facilities. 

During a multimedia conference, voice (or other au- 
dio) signals pass between conferees through conven- 
tional switch circuits and conference bridge circuits that 
are internal to the PSTN and that are administered by 
conventional conference management resources of the 
PSTN. However, data signals in the same conference 
(image data and other data) may be routed via a con- 
ference server over a public data network external to the 
PSTN, like the Internet or Web. Each such conference 
server is administered and controlled by the PSTN and 
has links to conference connection and conference 
management resources of the PSTN, such that a con- 
ference participant making a voice presentation can fur- 
nish image data to the conference server during or prior 
to the presentation, have the data stored in the confer- 
ence server, and then have the data uploaded from the 
conference server to the other participants in precise co- 
ordination with parts of the voice presentation referring 
specifically to respective display images. Connections 
between conference participants and public data net- 
works linked to a conference server may be dialled (tone 
keyed) telephone system connections or direct connec- 
tions through e.g. privately maintained lines and/or net- 
works. 

Each conference server and its aforesaid links with- 



in the PSTN is also useful to allow conferees to control 
operational services within both the PSTN and the ex- 
ternal data network, both before and during a confer- 
ence. 

5 In respect to coordinated transfer of image data, a 
conference server storing data furnished by a confer- 
ence participant can be requested by that participant to 
distribute the data toother conference participants. The 
request can be sent through either the external data net- 
to work (e.g. as a data signal representing the request) or 
through the PSTN conference connection as spoken 
commands if the PSTN is suitably equipped for detect- 
ing and reacting to such commands. 

In respect to user control of conditions or services 

is internal to the PSTN, such control can be used to inter- 
rupt a speaker in the conference; e.g. by a request over 
the data network causing the conference server to have 
images displayed to conferees (including the speaker) 
modified to show an indication requesting interruption 

20 of the speaker (e.g. a displayed indication connoting 
QUESTION TO ASK or NEED TO TALK); thereby allow- 
ing for orderly interruption of speakers without having 
plural participants trying to speak at the same time. 
Another use of this facility would be to allow a con- 

25 feree to request the conference server to adjust ampli- 
tude or other characteristics of audio signals currently 
being received by that conferee; whereby, for instance, 
a hearing impaired conferee could raise the volume of 
their received audio without affecting audio volume re- 

30 ceived by other participants. 

Another use would be to allow a conferee to invoke 
a new "follow me" type of service during a currently ac- 
tive conference wherein e.g. the respective conferee's 
connection to the conference could be shifted/switched 

35 from a fixed line in the PSTN (e.g. at a respective office 
or residence) to a mobile radio link. Thus, a conferee 
could leave home or office to travel elsewhere without 
disconnecting from the conference and without affecting 
continuity of the conference. 

40 Another use, if the PSTN conference centre is suit- 
ably equipped with speech recognition apparatus and 
applications therefor, would be to allow a voice com- 
mand over the PSTN conference connection to be de- 
tected by the speech recognition apparatus and used to 

45 server to control adjustment of a service condition in ei- 
ther the voice or data path or even in both paths (e.g. 
by displaying an indication to all participants stating or 
meaning LET'S TAKE A BREAK). 

Another aspect of this arrangement is that the con- 

50 ference server can be configured and connected to sup- 
port inclusion in a conference of "voice-only" and "data- 
only" participants, in addition to "voice and data" partic- 
ipants (the latter speaking and listening to voice, and 
sending and receiving data). Thus, a "voice-only" par- 

55 ticipant (e.g. a telephone user either without a computer 
or without concurrent access to the data network) could 
listen and speak to other participants, and a "data-only" 
participant (e.g. with access to the data network but ei- 
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ther no telephone or no concurrent access to the PSTN 
for voice) could send and receive image data without 
hearing what is said about received data. 

Another aspect is that "voice-only" connections 
could be set up without requiring telephone system op- 
erator intervention; for instance by having a conference 
master (e.g. the party whooriginated the respective con- 
ference) provide prospective voice-only participants 
with a pre-arranged password recognizable by speech 
recognition apparatus associated with the conference 
switching centre in the PSTN, having each such pro- 
spective participant call a special number connecting to 
the speech recognition apparatus, and having the 
speech recognition apparatus and an associated soft- 
ware application transmit appropriate signals to confer- 
ence management facilities of the associated switching 
centre, upon detection of the spoken password, to have 
the latter establish an appropriate connection addingthe 
caller to a currently active conference. The associated 
software application also could be used to signal the 
conference server to modify the currently displayed im- 
age so as to indicate addition of another conferee, and 
if desired the new conferee's name. 

Another aspect of this arrangement is that it sup- 
ports conferencing relative to a variety of different cus- 
tomer facilities; thereby supporting transfer of data be- 
tween conferees using an external data network and 
other conferees having a data terminal linked to the 
PSTN without access to any external data network, as 
well as transfer of voice between conferees using an ex- 
ternal data network for voice transmission and other 
conferees relying exclusive V on the PSTN for that f unc- 

t,0n Another aspect is that this conference sewer ar- 
rangement could potentially make two-party multimedia 
conferences using the server more productive and effi- 
cient than comparable services using totally separate 
voice and data connections, and that the cost of such 
support could be small enough to allow for attractive 
pricing thereof to end users. 

These and other aspects, features, advantages, 
benefits, and uses of our invention will be further under- 
stood by considering the following description and 
claims. 

Brief Description Qj The Drawings 



Fiqures 1 and 2. 

Figure 6 is a chart providing an overview of how 
voice and data are handled in the system of Figures 3 
and 4 

5 Figure 6a is a chart elaborating on information given 
in Figure 6. 

Figure 7 is a flow diagram illustrating how the inven- 
tion can be used to coordinate distribution and display 
of image data with voice presentations. 
w Figure 8 is a chart showing how the invention can 
be used by a conferee to alter usage of or conditions in 
the voice path of a conference. 

Figure 9 illustrates how the procedure of figure 8 
can be used to interrupt a speaker. 
is Figure 10 illustrates how a similar procedure can be 
used to modify voice amplitude. 

Figure 11 illustrates a preferred embodiment of the 

invention. 



20 rwaited Description 

-I rwinhion of Terms 

The following is a brief description of terms to be 
25 used in this description and their intended meanings: 

• LAN (Local Area Network) is a data communication 
network, either public or private, serving end users 
within a small area (e.g. within a building or geo- 
graphically concentrated commercial enterprise) . 



30 



35 



40 



45 



Figure 1 schematically illustrates a prior art system 
for multimedia conferencing wherein data can not affect 
voice handling and voice can not affect data handling. 

Figure 2 is a block diagram illustrating PSTN for 
managing voice connections in the system of Figure 1 . 

Figure 3 schematically illustrates application of this 
invention to the system of Figure 1 . _ 

Figure 4 is a block diagram illustrating PSTN fac.li- ^ 
ties for the arrangement of Fig. 3. 

Figure 5 is a chart providing an overview of how 
voice and data are handled in the prior art system of 



WAN (Wide Area Network) is a data communication 
network serving a larger geographic area than a 
LAN The Internet and World Wide Web (hereafter, 
Web) are •species" of public WAN systems that ef- 
fectively span the world, and serve members of the 
public at large having appropriate hardware, soft- 
ware and communication access thereto. 

PSTN (Public Switched Telephone Network) is the 
aggregate of all telephone communication resourc- 
es available to members of the public at large; in- 
cluding resources providing local, long distance 
land-based, and wireless/cellular telephone servic- 
es. 

Server means a communication-oriented computer; 
usually with fast internal clock, large memory, large 
storage capacity, and general capability of sustain- 
ing concurrent data communication with multiple 
end users. 

URL (Uniform Resource Locator) usually repre- 
sents the address of a specific document, file or dis- 
playable page image on the Internet or Web. It con- 
sists of a collection of characters identifying the da- 
ta its origin (e.g. the server containing it) and a spe- 
cific signalling protocol used for its transmission. A 
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URL may be used by "browsing" software, at loca- 
tions of Web end users, to designate a file to be re- 
trieved from a remote server on the Web. 

• Conference call (or session) represents a voice 
connection service provided by the PSTN wherein 
two or more end users are connected so that each 
can speak and the others can hear what is said by 
the speaker. Participants in a conference call are 
herein termed participants, attendees, conferring 
parties or conferees. 

2. Explanation of Problem Presently Solved 

This invention concerns multimedia conferences 
wherein conferees concurrently may converse and ex- 
change data, the latter including displayable text and 
graphic images. In today's facilities for multimedia trans- 
mission, data and voice are routed over separate paths 
such that it is difficult if not impossible to have presen- 
tations of image data coordinated with vocal presenta- 
tions in a coherent manner. 

As exemplified in Figure 1 , a typical contemporary 
(prior art) facility of this kind routes voice signals exclu- 
sively through the PSTN, and data signals exclusively 
through a parallel public data network like the Web, but 
without any possibility for interaction between the two 
networks; e.g. to allow for coordination of display image 
presentation with vocal presentation, or to allow for data 
signal communications to alter services currently effec- 
tive in the PSTN, etc. 

The PSTN, shown at 1 , has conference call man- 
aging and administering facilities 2 which oversee voice 
signal connections between conference participants but 
have no tie-in to simultaneous data transmissions be- 
tween the participants. In figure 1, A and B represent 
two (of two or more) end-user stations currently having 
a conference connection on the PSTN. Equipment used 
at these stations are indicated respectively at 5 and 6. 
Station A's equipment 5 includes a telephone 7 and 
computer/display apparatus 8, and station B's equip- 
ment 6 includes a telephone 9 and computer/display ap- 
paratus 10. Telephones 7 and 9 are connected through 
transmission lines 11 in the PSTN and (not shown) 
switches administered and controlled by the PSTN. Tel- 
ephones 7 and 9 are not locally connected to respective 
computers in this illustration, but even if so connected 
neither the PSTN nor the Web could act to coordinate 
or otherwise inter-relate voice and data transmissions; 
i.e. only the end users of these stations could act to co- 
ordinate data transfers with voice, by agreeing on a set 
of protocols beforehand, and such actions could not af- 
fect services provided in either network. 

To exchange data in the typical arrangement of Fig- 
ure 1 , conferees at stations A and B link to a server 1 2. 
The data is transferred e.g. through modems indicated 
at 1 3 and 14, transmission line paths shown collectively 
at 15, and a data communication network such as the 



Web shown at 16. In the illustrative example of this fig- 
ure, server 12 and modem 13 are shown as locally ac- 
cessible to station A and other not-shown stations via a 
LAN 17, and remotely accessible to station B via the 
5 Web 1 6 and modem 1 4. Thus, in the illustrated example, 
station A and the not-shown other stations could be 
computer stations within a building or a group of geo- 
graphically proximate buildings (e.g. a building or build- 
ings locally serving a business enterprise), and server 

1 2 and modem 1 3 could be shared via LAN 1 7 by occu- 
pants of the respective building(s). Thus, the occupants 
would receive shared data handling and modem con- 
version services, rather than requiring separate data 
handling and modems at their individual offices or sites. 
In contrast, modem 14 is used only by station B and is 
within a few feet of the computer at that station. 

In this configuration, modems such as 13 and 14 
are required only for linkage to the applicable data net- 
work such as the Web. Also, server 12 can be located 
anywhere within communication reach of the respective 
set of stations that it serves (i.e. a WAN could be sub- 
stituted for LAN 17 if necessary and/or appropriate). 
Futhermore, it should be understood that server 1 2 may 
be only one of plural servers through which the confer- 
ees exchange data. Although lines 1 5 between modems 

13 and 14 are shown as passing through both the Web 
16 and PSTN 1 , in this example they have no connecta- 
bility to the call management facilities 2 of the PSTN. 

Referring to Figure 2, the prior art call management 
facilities 2 include a call management system 20 (e.g. a 
computer system or the like) which manages and ad- 
ministers conventional switching apparatus 21 and con- 
ventional bridging apparatus 22 to establish and termi- 
nate conference call connections. Apparatus 22 com- 
prises facilities allowing for voice signals to be amplified, 
summed and distributed among the conferees. 

In this environment, functions and end user servic- 
es provided by the Web are not affectable by voice sig- 
nals in the PSTN, and conditions in the voice connection 
path are not affectable by signals sent through the data 
connection/Web. 

Figure 5 shows how a conference session is estab- 
lished and maintained in the prior art environment 
shown in Figures 1 and 2. As shown at 23, each partic- 
ipant establishes separate linkages to the PSTN and to 
a server such as 1 2 in the data network/Web, the former 
exclusively for voice contact and the latter exclusively 
for data interchange. Therefore, each participant re- 
quires simultaneous access to the PSTN and the data 
network via at least two physically or logically separate 
lines. Typically, in this environment, arrangements for 
the conference are made in advance and may require 
assistance of telephone system operators to link indi- 
vidual callers to the conference and announce their 
presence to those already participating. Usually, the ad- 
vance arrangements define a maximum number of at- 
tendees, and once that number is reached no additional 
participants are permitted (unless and until one or more 
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current attendees drop out of the conference). 

As suggested at 24, this environment provides two 
concurrent but uncoordinatable conferences, one han- 
dling speech between the participants via voice connec- 
tions within the PSTN, and the other handling movement 
of displayable data between them via the Web and serv- 
ers such as 12. Such servers may be administered by 
either the Web or entities that are not part of either the 
Web or the PSTN, and they may have security facilities 
restricting external access to them which are costly to 
acquire and maintain. If the participants have facilities 
for collaboratively modifying displayed images, all must 
have corresponding non-standardized software appli- 
cations permitting that collaboration. 

Since the voice and data connections are separate 
and not inter-linkable, signals in either connection path 
can not affect services performed in the other path. 
Thus, data signals sent over the Web can not be used 
to prompt the PSTN to alter its current handling of the 
conference (e.g. to increase reception volume of a hear- 
ing impaired participant, or modify the connection of a 
participant from e.g. a fixed residential or office phone 
to a mobile radio unit), and voice signals can not be used 
to control or coordinate presentation of image data to 
the participants; e.g. in coordination with a voice pres- 
entation. 

The present invention seeks to resolve these and 
other problems or shortcomings of existing multimedia 
conferencing arrangements, by incorporating confer- 
ence serverfacilities into the PSTN which are effectively 
linked to both data network(s) such as the Web and the 
PSTN. These conference servers are communication- 
oriented digital computer systems having communica- 
tion links to both the PSTN and public data communica- 
tion networks like the Internet, Web, etc., and their func- 
tions and sites of location are administered by the 
PSTN. The conference servers in general will be able 
to communicate with conference management resourc- 
es of the PSTN. They also may be able to communicate 
with speech recognition apparatus and associated ap- 
plications within a conference switching centre of the 
PSTN so as to be able to react to voice commands is- 
sued by conferees; for example, commands to distribute 
data representing specific display images provided by 
the conferee issuing the command. These servers and 
their integration into the PSTN would enable the PSTN 
to offer cost-effective conference management services 
to its end users that are either completely new or pres- 
ently considered impractical for existing PSTN facilities. 
Such services would include cost effective handling of 
voice conferences over conventional (unleased) lines in 
the -plain old telephone system" (POTS), concurrent 
with distribution of data over public data networks par- 
allel to the PSTN (e.g. the Web), wherein conferees 
would be able to use their data network connections to 
control conditions of operation in the voice connection 
path, and conversely able to use voice commands to 
control functions performed in the data network, for ex- 



ample, to control distribution of image data in coordina- 
tion with voice presentations. Thus, a conferee would 
be able to administer and/or invoke specific services in 
either the PSTN or the parallel data network via signals 
s sent on the other network. An example would be that a 
conferee could issue a "follow me" data request to 
switch their current voice connection (e.g. from a sta- 
tionary phone to a mobile radio one), or another data 
request to have the amplitude or other parameters of 
10 voice transmitted to them changed (e.g. to adjust the 
sound delivered to a hearing impaired conferee). 

Although conference calls on the PSTN heretofore 
usually have involved more than two participants, with 
our inventive arrangement a conference call may be 
is used as an efficient method for two parties to converse 
while concurrently exchanging data (the term 'efficient 
method" is used here to imply a process that could be 
more cost-effective and productive than a simple voice 
connection over the PSTN concurrent with a simple and 
20 completely separate data connection over the Web or 
another line path within the PSTN). 

Those skilled in the art will recognize that the ar- 
rangement of Figures 1, 2 and 5 is not the only previ- 
ously known technique for sustaining concurrent multi- 
25 media conferences. For instance, presently known soft- 
ware applications for the Internet/Web allow users of 
those networks and respective applications to effective- 
ly carry on "long distance" voice conversations using on- 
ly local resources of the PSTN to link the users to locally 
30 accessible servers. These applications however have 
generally tended to provide voice transfer qualities infe- 
rior to (albeit cheaper than) equivalent long distance 
voice connections on the PSTN; generally due to the 
limited bandwidth available within public data networks 
35 today for such communications, and indefinite routing 
latencies (delays) of those networks. 

There are also applications purporting to provide in- 
tegrated handling of voice and data over data networks 
like the Web, as well as conferencing services in respect 
40 to such handling. But they too have limitations in respect 
to quality due mainly to bandwidth limitations deriving 
from the nature of today's network infrastructures and 
from the integrated handling of voice and data through 
common transmission paths. Furthermore, such ar- 
45 rangements are incapable of enabling their users to ef- 
ficiently coordinate distribution of displayable data with 
voice presentations of respective users. 

It is also possible for users to converse and transfer 
data concurrently over leased lines, but such lines would 
50 be considerably more costly than the type of conference 
services presently contemplated. 

The present invention seeks to provide cost-effec- 
tive conferencing of voice and data such that: 1) neither 
process is degraded by bandwidth limitations; 2) trans- 
55 mission of data between conferees can be coordinated 
efficiently with voice remarks of the conferees; and 3) 
services within either the PSTN or a parallel data net- 
work such as the Web can be efficiently administered 
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and/or invoked by conferees during a conference. 

3. The Present Solution 

An overview of how the present invention solves the 
foregoing problems is presented in figures 3, 4, 6 and 

6a. 

Figure 3 shows the arrangement of Figure 1 modi- 
fied in accordance with our invention. Elements func- 
tionally identical to those in Figure 1 are denoted with 
identical numbers. Consequently this shows that novel 
elements of the invention are in the conference control 
centre 2a and are associated at least in part with the 
data path represented by lines 28, 28-a. 

Figure 4 shows that conference centre 2a includes 
a conference management system 20a and a confer- 
ence server system 30. Conference server 30 is a com- 
munications adapted computer system maintained and 
administered by the PSTN. It's loose integration with call 
management resources of the PSTN, including its con- 
nection 31 to conference management facilities such as 
20a, enable it to control services performed relative to 
an existing PSTN conference connection, in response 
to signals sent over a path external to the PSTN; exem- 
plified by the indicated external connection(s) 32 to the 
Web (these connections being viewable as extending to 
line 28-a in Figure 3). 

These external connections permit new conferees 
to easily be added to an active conference without as- 
sistance of telephone company operators. Such added 
conferees may be pre-scheduled to join the conference 
at its inception, or they may join on an ad hoc basis, and 
their additions may be displayed to prior participants 
eliminating need for announcements by assisting tele- 
phone operators. They also enable conferees to easily 
and economically control other PSTN conference-use- 
ful services (either before or during a conference) with- 
out operator assistance; e.g. to vary connections of par- 
ticipants moving between fixed desk and mobile cellular 
telephones (in a manner further discussed later on). 

Conference management system 20a differs from 
"prior art" conference management system 20 of Figure 
2 primarily in its connection 31 to conference server 30, 
and in functions and services derivative from that con- 
nection. As discussed later, speech recognition appara- 
tus and associated applications within the PSTN may 
be linked to conference bridge 22, for recognizing voice 
commands of conference participants and enabling ei- 
ther management system 20a or the data network, or 
both, to perform actions associated with such com- 
mands; e.g. real time actions instantly affecting condi- 
tions in either the voice connection path, the data trans- 
mission path, or both. 

It will also be shown later that although participating 
stations in the arrangements of Figures 1 and 3 have 
separate telephones and computers, as well as sepa- 
rate communication paths for each, the conference cen- 
tre arrangement of Figure 4 would easily be adapted to 



serve conferees having only a telephone, or only a com- 
puter, or conferees having both a telephone and com- 
puter operating through a single physical line connect- 
ing to either the PSTN or the Web. Adjustments needed 

5 to support these reduced configurations are discussed 
later in this description. Furthermore, although station 
equipment external to the PSTN and Web is shown in 
Figure 3 as identical to that in Figure 1 , it should be un- 
derstood that this is done only to facilitate comparison 

10 of the present inventive solution to prior art. Therefore, 
it should be understood, and will become apparent from 
later parts of this description, that stations having many 
different forms of equipment are supportable by the 
present arrangement. 

is Figure 6 provides an overview of capabilities afford- 
ed by the present conference server arrangement. 

As indicated at 36, conference participants in both 
voice and data aspects of a conference still establish 
parallel voice and data connections over physically or 

20 logically separate lines extending to the PSTN and data 
network. However, as indicated at 37, data transferred 
between the conferees is funnelled through conference 
server 30. Consequently, servers external to the PSTN 
may not be required for presently contemplated confer- 

25 ence services, except perhaps for providing data collec- 
tion, storage and distribution services within e.g. private 
enterprises (office buildings, plants, etc.). In other 
words, end users at self-contained participating stations 
like station B, having unshared data handling and mo 

30 dem facilities, need only link to PSTN conference serv- 
ers such as 30 and need not have any direct links to 
other servers such as 12. 

As suggested at 37, since server 30 has direct links 
to conference management facilities of the PSTN, im- 

35 age data transmissions on the Web are coordinatable 
with vocal aspects of the conference, and other data 
transmissions may be used by participants to alter or 
invoke services affecting their voice connections (e.g. 
volume of voice signals sent to them, interruption of 

40 speakers, switching phone connections e.g. from fixed 
lines to mobile radio links, etc.). Block 37 also indicates 
that data security may be provided by the conference 
server at "relatively small cost"; since that server and its 
functions are effectively shared by all users of the re- 

45 spective PSTN conference management and bridging 
facilities, rather than by potentially fewer users of (pos- 
sibly a larger number of) servers such as 1 2, whence its 
security functions may be apportioned among a poten- 
tially larger population of users than that of servers such 

50 as 12 external to the PSTN. This should be more fully 
understood from the description below of Figure 11 . 

Figure 6a elaborates on functions indicated in Fig- 
ure 6. Voice signals flow only through switch and con- 
ference bridge connections in the PSTN (block 40). Data 

55 signals flow only through data connection paths be- 
tween the conference server and conferees, generally 
through the Web, and generally over connection paths 
separate from the voice connection (block 41). Voice 
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commands/requests may be detected b speech ^ecog 
nition apparatus in the PSTN, and used in as soc aated 
control applications to initiate controlling actions by e.- 
the the conference manager, the conference server o 
both (block 42); i.e. actionsaffecting conditions .neither 
the vie path, the data path or both. Data passed 
Lough the data network includes data repressing 
dispteyable images, and conferee requests and other 
control signals such as coordinate cues (block 43J. 
SLs may be signalled to the conference server by a 
co^ferefand handled by the conference server to ap- 
pt a r as displayed icons or other viewable features at 
Sons of other conferees (also block 4* Requests 
may be forwarded to the conference manager for han 
Sing if the associated functions are not d.rectly controU 
lable bythe conference server.such as f unctions requ*- 
ing changes of conditions in voice connects circurts 

^u'ho'uld be noted here that this invention contem- 
plates that requests for changes in voice connects c.r- 

cuits may be sent to the conference server erther aver 
a data network such as the web or a third paral lei com- 
munion network, such as a network handling two 
way pagercommunicationsfor conferees equipped wrth 

^ SreTiiuTrates how data representing display- 
able images (e.g. charts) may be presented to all con- 
SeesTcoordination wrth a voice presentation by one 
of the conferees. The image data is sent by aconference 
master or moderator (e.g. a party ong.nating the confer- 
ence a party owning or controlling the data, a party 
scheduled to make the voice presentation) to the con- 
ference server, either before or during j£ , ^ference 
and stored by the conference server (block 50). There 
after the data is distributed by the conference server, 

en by the voice presenter (block 51). Images stored in 
"he conference server may be distributed to the cor, er- 
ees one at a time, for immediate display upon recep K£ 
and each page may be sent in response to a cue from 
fhe mlster/mlderator. This cue may be manrtested e - 
ther as data representing a request to have the confer- 
ence server distribute a specified chart (or a chart next 
fn a predetermined sequence, etc.), or as an equivalent 
voice command picked by speech recognrt.on appara- 
tus linked to the voice connection of the conference and 
transferred to the conference server. 

Figure? indicates furtherthat conferees hav.ngcol- 
laboration software (software enabling them to coltebo- 
ratively edit a displayed image) may make edrting 
changes to a currently viewed page and have he chang- 
es communicated to the other conferees via the confe - 
ence server (block 52). Such changes may be commu- 
nicated in any of several ways, either as transmissions 
indicating only the edited changes, or as transmissions 
including'data representing the whole page w h the ed- 
ited changes, or in any other manner cons.stent with the 
intended communication effect. 



Collaborative editing may be achieved in several 
wavs The method preferred presently would invoke 
useatpanicipatingstationsofbrowsersoftvvareoapable 

of receiving and executing mini-programs or applets 
s su STLks required for performing ootabort*. 
modifications. Such applets may for instance be wntten 
Tn a commonly used language such as the Java™ lan- 
guageTuava is a Trademark of Sun Microsystems com- 
£3 which is intertable by a number ^o existing 
,o browsers constructed to interpret Java scripts (e^g Net- 
scape Navigator, Microsoft Explorer, etc), stored in the 
cSCnceserver.andtransferredfromtheconference 

se^er to conference participants as the latte, s^v onto 
L data network for a conference session. These ap- 
,5 Suld be constructed to either 

at participant stations when a conference ends, or to e 
ma^SJat participant stations after a conference tor 
use in future conferences (since the server wou d not 
easily distinguish which stations have previous* - rans- 
20 ferred applets, the preferred method would be to d.stnb- 
Se them as a conference begins and have them setf- 
olstnTcTat partteipant states when the conference 
ends Such applets could enable conferees to modify 
graphic and/or text elements of a currently displayed 
2S age, and communicate modifications to the con- 

ference server for distribution to stations of other con- 

f6re Aless preferred method would be to require all par- 
ticipants to have particular proprietary software (e^g. Lo- 

dteplayable documents within a special commun.cat.on 
environment required by that software. 

TLwn at 53. conferees requiring special servic- 
es during a conference, that are invocable through the 
es oLconnectton, send data representing appropnate^- 
quests to the conference server, and the latte ^reacts 
appropriately to have the requested f unction performed 
frt Sable. Box 53 also shows that conferees may 
send f ler data' (data other than edited images ongi- 
40 na^bTtheconfefencemasterandrequeststorse^ 
changes) and that such other data may be d.stnbuted 
to the other conferees if appropriate. 

Figure 8 illustrates an example of one such speaal 
service request for action pertinent to the voice confer- 
« ence. Aslown at 55. data representing the request* 
sent to the conference server. As suggested at 56 the 
server determines if the requested service is avartable 
and either takes action suited to the request d.rectfy or 
signals the conference manager to do so. depending up- 
50 on where the requested action is executable. As seen 
at 57 the server then returns to the requestor eithe an 
acknowledgement of the request or an indicate that 
the requested service is unavailable. 

Figure 9 gives an example of a data request per- 
55 tainingtoconditionsinthevoicecircurts.lnthisexample. 
heequestorseekstointerruptaparty currently speak- 
inq At 60, the request is sent to the conference server, 
and at 61 the server responds by illuminating an icon on 
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the image currently displayed to the participants; e.g. an 
icon representing ASK QUESTION, LET ME TALK, etc. 
This may be done either by signalling browsers at par- 
ticipant stations to modify the portion of the displayed 
image containing the icon, or by signalling that the cur- 
rent image is ■stale" and requires reloading (in which 
case the browsers would request reloads and receive 
the current image with the icon illuminated), or by any 
other action resulting in an appropriate visual cue to the 
speaker that another party wants to speak. 

Figure 10 illustrates another example of action per- 
taining to the voice conference. Here, the request (op- 
eration 64) is to alter a characteristic (e.g. volume) of 
voice signals that are being sent to the requestor (e.g. 
a hearing impaired party). In this case, the conference 
server forwards a corresponding request to the confer- 
ence manager (operation 65), and the latter takes action 
66 relative to the voice path (conference bridge, etc.) to 
effect the desired change. A similar request action could 
be used to have the conference server and conference 
manager change the phone line connection of a re- 
questing participant; e.g. to switch that participant's link 
to the conference from a fixed line (residential or office) 
to a mobile radio link. In this last example, the switching 
action could be implemented by having the conference 
manager first communicate with a mobile switching of- 
fice to establish the mobile radio link to the conference 
bridge, and then act to sever the connection of the re- 
questor's fixed line. 

Figure 11 schematically illustrates a conference 
control system, in accordance with the present inven- 
tion, which shows the different configurations of user 
stations that are supportable and how such support is 
given. As such, this may be considered as showing the 
best mode contemplated for carrying out our invention. 

This figure shows four user stations - stations A, B, 
C and D - having various different configurations of 
equipment which for the following discussion are as- 
sumed to be actively connected, via conferencing centre 
70 in the PSTN, in a multimedia conferencing session 
(i.e. users of these stations are participants in that ses- 
sion). 

Station A contains only a single analog telephone 
72; i.e. its user is a voice-only participant in the confer- 
ence. Station C contains an analog telephone 74, digital 
computer 76 and modem 78, with the telephone con- 
necting to conference centre 70 in a first connection path 
within the PSTN that is described below, and the com- 
puter and modem connecting in a physically separate 
second path within the PSTN, also described below; i. 
e. its user is a participant in both voice and data ele- 
ments of the conference, whose voice and data flow 
through physically different paths within the PSTN. Sta- 
tion D contains only a computer 80, modem 81 and 
speaker 82 and microphone 84 both attached to the 
computer; i.e. its user is a voice and data participant 
whose voice and data are constrained to pass through 
the respective computer 80 and modem 81 , and there- 



fore have only a single physical path of connection to 
the conference centre (which path, in the illustrated ar- 
rangement, is entirely within the PSTN). Finally, station 
B contains an analog telephone 85 and digital computer 
88 which connect to the conference centre through 
physically separate paths described below, one in the 
PSTN (the voice path) and the other external to the 
PSTN (the data path). 

Computers at these stations (those other than sta- 
tion A) are shown as laptop style personal computers, 
but it is understood that they may be computers of any 
physical form (e.g. laptop, desktop, floor console, or 
even computers integral to an appliance such as a TV 
monitor), and need not even be personal type comput- 
ers (i.e. they could even be computers like mainframes 
or such that are larger in size and operating capabilities 
than personal computers). The only requirement is that 
they contain browser software that is compatible with 
applets or scripts written in a common language such 
as Java, and that communicate with protocols that are 
compatible with those employed at the conference serv- 
er to be described below. 

Conference centre 70 comprises call connecting 
circuits 90 containing switching circuits needed for sim- 
ple non-conference connections as well as bridging cir- 
cuits required for conference connections. The size and 
capabilities of this element are dependent upon the an- 
ticipated peak traffic in non-conference and conference 
calls for the respective conference centre. 

Conference centre 70 also comprises a call man- 
agement facility 92, a conference server facility 93, an 
administrative facility 94, an (optional) speech recogni- 
tion facility 95 (for speech recognition applications in- 
cluding those relevant to conference management), and 
an (optional) archive facility 96 (for voice and data ar- 
chiving applications that could e.g. allow voice and data 
portions of a conference to be replayed to conference 
participants at some later date). Facilities 92, 93, 95 and 
96 have direct connections to connecting circuits 90 for 
handling signals to and from end user stations. All of the 
facilities 92-96 connect to a LAN 1 00 for communicating 
with each other. The connection circuits 90, facilities 
92-96 and LAN 100 are all components of conference 
cents 70, and the latter is a component of the PSTN. 

Telephone 72 in station A connects to connecting 
circuits 90 through a part of the PSTN indicated at 104. 
Telephone 74 in station C connects to circuits 90 through 
a part of the PSTN shown at 1 06, while computer 76 
and modem 78 in the same station connect to confer- 
ence server 93 through another part of the PSTN 108 
and a modem bank 109 (i.e. this station has two physi- 
cally separate lines locally connecting to the PSTN and 
having voice routed through the switch circuits 90 and 
data routed to the conference server over a path parallel 
to and separate from circuits 90). The computer 80 and 
modem 81 in station D connect to modem bank 109 via 
a link to PSTN portion 108 that is physically separate 
from that of modem 78 in station C. Consequently, the 
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voice traffic associated with speaker and microphone at- 
tachments 82 and 83 in station D flows through compu- 
ter 80, modem 81 , PSTN portion 1 08. and modem bank 
109 interspersed with any other data issued by or di- 
rected to computer 80; and all voice and data signals 
routed between modem 81 and PSTN portion 108 are 
in a common (analog) form. 

Modem bank 1 09 serves to convert signals between 
analog and digital forms. Signals sent by modems 78 
and 81 are converted to (baseband) digital form by mo- 
dem bank 109 and passed to conference server 93 in 
the latter form. Signals sent by conference server 93 to 
modems 78 and 81 (and their respective computers) are 
converted in bank 109 from (baseband) digital form to 

analog form. „ . 

Telephone 85 in station B connects to circuits 90 via 
portion 110 of the PSTN, while computer 88 in the same 
station connects to conference centre 70 through a path 
112 external to the PSTN. Path 112 comprises a LAN 
114 connecting computer 88 and other (not-shown) 
computers to conference server 93 through a gateway/ 
firewall security element 115 and a portion 116 of the 
Web (or other data network). 

Notice that the connections between conference 
server 93 and computers 78 (station C) and 80 (station 
D) are entirely within the PSTN (through modem bank 
109 which is a component of conference centre 70 and 
PSTN portion 108), whereas the connection between 
the server and computer 88 is external to the PSTN (i. 
e through Web portion 116, and private network 114)^ 
Notice also that signals sent between modem bank 109 
and computers 78 and 80 are analog in form, whereas 
those sent between server 93 and the modem bank are 
in digital form and those sent between server 93 and 
computer 88 are in analog form in transit through the 
Web and in baseband digital form in transit through LAN 

114 and firewall 115. Signals passing between firewall 

1 1 5 and the Web are converted between digital and an- 
alog forms through not-shown modems and other cir- 
cuits located at the access node of the Web at which 
signals from the firewall pass into the Web and signals 
leaving the Web transfer to the firewall. 

An optional direct connection between modem 81 
and web portion 116 issuggestedat 117, to indicate that 
the present conference server arrangement can be 
used to accommodate voice and data transmission 
through the web, and link voice portions thereof to voice 
conference facilities within the PSTN. 

Conference server 93 has a voice only connection 
(V) to switch circuits 90 and a voice and data connection 
(V/D) to the same circuits. These connections are re- 
quired for routing some of the conference voice and data 
traffic between the participants, where the respective 
traffic signals are required to pass between connection 
circuits 90 and either modem bank 109 or the parallel 
datanetwork 114-116 linked to station B. The voice onty 
connection is used e.g. for routing analog voice signals 
through the server, between telephones 72 (station A), 



74 (station C) and 85 (station B). The voice and data 
connection is used e.g. for routing analog voice signals 
through the server, between the speaker and micro- 
phone attachments to computer 80 at station D and any 
5 of the telephones at stations A, C and D, as well as for 
routing analog data signals exclusive of voice between 
computer 80 and the other station computers 76 (station 
C) and 88 (station B). 

It should be noted that interspersed voice and data 
10 signals passed from station D to the conference server, 
via modem bank 109, usually will have to be separated 
at the server 93 to permit the voice signals to be trans- 
ferred via circuits 90 to the telephones at the other sta- 
tions, while the data is subjected to other handling for 
is example, storage in the server, or interpretation by the 
server as commands, cues or requests, etc.). Similarly, 
voice and data transferred to station D would have to 
flow through the modem bank and would have to be ap- 
propriately merged into a common form suitable for han- 
20 dling by modem 81 at station D. These separation and 
merger functions are state of the art logical functions 
that can be performed either in the modem bank by suit- 
able logic, or by the server 93 if the data is in a form 
suitable for reading by that entity. 
25 It should be understood that in conducting confer- 
ence voice and/or data signals from conference connec- 
tion circuits in the PSTN to external networks such as 
the Web. the present conference server would receive 
the respective signals along with an indication identrfy- 
ao ing the associated conference and route the respective 
signals to appropriate external paths applicable to the 
associated conference. Conversely, in conductingvo.ee 
and/or data signals from an external source to the con- 
ference connection circuits in the PSTN, the conference 
35 server would receive the respective signals, recognize 
the conference associated with their origin, and transfer 
the signals to the PSTN connection circuits along with 
an identification of the associated conference, causing 
the signals to be routed over telephone company circuits 
40 and lines to destinations appropriate for that confer- 
ence. 

It should be recognized that the foregoing arrange- 
ment would be readily adaptable to perform any or all of 
the following functions: 

45 

. add conferees to an existing conference (either pre- 
determined individuals or others entering on an ad 
hoc basis) without telephone operator assistance; 



50 



55 



announce entry of new conferees via computer dis- 
plays, also without operator assistance 

drop conferees from an existing conference without 
operator assistance; 

split a conference into separate sub-conferences 
involving discrete sub-groups of the group partici- 
pating in the parent conference; 
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• selectively block portions of a conference (voice 2. 
and/or data) from presentation to some partici- 
pants; e.g. to present confidential data to partici- 
pants authorized to receive such and block it from 
participants not authorized to receive it. s 

The foregoing and other aspects of the invention will 
be further appreciated from the following claims. 

10 3. 

Claims 

1. A conference server system, for use in a public 
switched telephone network (PSTN) having confer- 
encing circuits for routing analog signals, including 15 
voice signals, between end users of said network, 
and conference management facilities for manag- 
ing operations in said conferencing circuits to pro- 
vide conference connections between end users re- 
quiring said connections; said conference server 20 
system comprising: 

a computer system for storing and processing 
data; 

means connecting said computer system to a 25 
data network external to said PSTN for trans- 
ferring data between said computer system and 
end users of said PSTN currently participating 
in a voice conference utilizing said conferenc- 
ing circuits; said data being transferred in par- 30 
allel over said data network while signals rep- 
resenting voice utterances of said participating 4. 
end users are transmitted through said confer- 
encing circuits; said data including displayable 
image data and data representing functional re- 35 
quests and commands issued by individual 
said participating end users and requiring at- 
tention by said conference server system; said 
functional requests including requests from a 
said participating end user to said conference 40 
server system to have said conference server 
system distribute displayable image data, pre- 5. 
viously furnished to said conference server sys- 
tem by the participating end user issuing the re- 
quest, to others of said participating end users; 
and 

means linking said computer system to said 
conferencing circuits and said conference man- 
agement facilities for enabling said computer 6. 
system to coordinate distribution of said dis- so 
playable image data between said participating 
end users in coordination with voice inter- 
changes between said participating end users; 
said linking means also being useful for ena- 
bling said computer system to participate in sat- ss 
isf action of functional requests and commands 
issued by said participating end users and said 
conference management facilities. 



A conference server system in accordance with 
claim 1 wherein said means for connecting said 
computer system to said conferencing circuits and 
conference management facilities includes: 

means responsive to certain said functional 
requests for signalling said conference manage- 
ment facilities to vary conditions of operation in said 
conferencing circuits. 

A conference server system in accordance with 
claim 1 or 2 wherein said PSTN includes: speech- 
recognition means linked to said conferencing cir- 
cuits for detecting spoken commands and function- 
al requests uttered by said participating end users; 
and 

means linking said speech-recognition means 
to said conference server computer system for 
enabling said conference server computer sys- 
tem to participate in satisfaction of said spoken 
commands and functional requests; and 
wherein said conference server computer sys- 
tem comprises: 

means responsive to signals received from said 
speech-recognition means for distributing indi- 
vidual pages of said displayable image data to 
said participating end users and for performing 
other functions relevant to said voice confer- 
ence in which said end users are currently par- 
ticipating. 

A conference server system in accordance with 
claim 2 wherein said signals to vary conditions in 
said conferencing circuits are useful to cause the 
telephone connection of a said participating end us- 
er to be changed so as to enable the respective end 
user to change between stationary and mobile 
phones separately connected to said PSTN without 
interrupting continuity of a currently active confer- 
ence. 

A conference server system in accordance with 
claim 2 wherein said signals to vary conditions in 
said conferencing circuits are useful to effectively 
vary the volume of sound currently being delivered 
to a telephone receiver used by one of said partic- 
ipating end users. 

A conference server system in accordance with any 
preceding claim wherein said connections between 
said computer system and said conferencing cir- 
cuits are useful to enable said computer system to 
transfer signals representing voice and data origi- 
nated by said participating end users between said 
participating end users, and to enable said compu- 
ter system to route said voice signals to said con- 
ferencing circuits and said data signals to said data 
network; whereby participating end users lacking 
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connections to said data network can send data 
over the PSTN, and have said data routed through 
said data network by said server system to other 
participating end users that have connections to 
said data network. 

7 A conference server system in accordance with any 
preceding claim wherein said external data network 
is a public data network, like the Internet or World 
Wide Web, that is accessible to any member of the 
public complying with requirements of the respec- 
tive network. 

8 A conference server system in accordance with any 
preceding claim wherein said conference server is 
effective to form a circuit path between said confer- 
encing circuits in said PSTN and said external data 
network for selectively routing signals representing 
both voice and data between said conferencing cir- 
cuits and said external data network. 

9 A conference server system in accordance with 
claim 8 wherein said circuit path formed by said con- 
ference server enables said PSTN and external da- 
ta network to support conferences between confer- 
ence participants having different equipment and 
different communication connections including but 
not limited to any and all of the following: 

a participant having a conventional telephone 
linked to the PSTN and a computer separately 
linked to said external data network; 
a participant having only a conventional tele- 
phone linked to the PSTN; 
a participant having only a computer linked to 
said external data network; said computer un- 
equipped for handling voice; 
a participant having only a computer linked to 
said external data network; said computer be- 
ing equipped for handling both voice and data 
transmissions relative to the respective partic- 
ipant; 

a participant having only a device linked to said 
external network; said device being equipped 
only for handling voice communications relative 
to the respective participant; and 
a participant having only a computer linked to 
said PSTN; said computer being equipped for 
handling both voice and data transmissions rel- 
ative to said PSTN. 



conference management facilities is also effective 
to link said conference server computer to said 
speech recognition means. 

s 11 A conference server system in accordance with 
claim 10 wherein said means linking said confer- 
ence server computer to said conferencing circuits, 
said conference management facilities and said 
voice recognition means comprises a network inter- 
10 nal to said PSTN. 

12 A conference server system in accordance with 
claim 11 wherein said internal network is a local ar- 
ea network (LAN). 
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10. A conference server system in accordance with any 
preceding claim wherein said PSTN includes 
speech recognition means connected to said con- 
ferencing circuits for responding to voice com- 
mands uttered by said conference participants and 
said means linking said conference server compu- 
ter system to said conferencing circuits and said 



A conference server system in accordance with any 
preceding claim wherein said PSTN includes a mo- 
dem bank forming signal transmission paths inter- 
nal to said PSTN but parallel to said conferencing 
circuits and said conference server includes: 

means linking said conference server compu- 
ter to said modem bank for routing voice and data 
signals between said conferencing circuits and said 
modem bank and between said external data net- 
work and said modem bank. 

14. A conference server system in accordance with 
claim 13 wherein said conference server and mo- 
dem bank are effective jointly to: 

separate voice and data signals received in 
said modem bank from a single said participant; 
route the separated voice signals to said con- 
ferencing circuits for connection to telephones 
of all participants having telephones; and 
selectively route the separated data signals to 
said conferencing circuits, for delivery to partic- 
ipants having computers directly connected to 
said PSTN, and to said external data network 
for delivery to participants having computers di- 
rectly connected to said external data network. 

1 5. A conference server system in accordance with any 
preceding claim wherein said conference server 
computer system and conferencing circuits are 
jointly effective to allow any of the following actions 
to be implemented, without telephone operator as- 
sistance, relative to said conference in which said 
PSTN end users are currently participating: 

• add predetermined participants to said confer- 
ence; 

• add new participants to said conference on an 
55 ad hoc basis; 

• selectively remove (disconnect) participants 
from said conference; 
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• display additions of new participants and re- 
moval of old participants as they occur; 

• permit said conference to be sub-divided into 
plural sub-conferences between subgroups of $ 
the participants in said conference; 

• block portions of a conference relative to a sub- 
set of its current participants (e..g. to prevent 
unauthorized participants for listening to and/or to 
viewing confidential information). 
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Participants establish separate conference 
connections to PSTN and a shared server 
having no link to PSTN conference 
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Participants converse over PSTN connection 
and exchange data via server. Since server is 
unlinked to voice conference, data interchange 
can neither be coordinated with nor affect voice; 
and data security may be costly to implement 
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Participants establish separate conference 
connections to PSTN and a conference server 
managed and administered by the PSTN 
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Participants converse over PSTN connection 
and exchange data via server. Since server b 
linked to and part of voice conference center, 
data interchange can be coordinated with voice 
presentations of individual speakers, and even 
used to affect conditions in the voice 
transmission path. Data security may be 
provided by PSTN at relatively little cost 
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Figure 6a 



Voice signals flow between 
conferees only through 
PSTN lines and circuits, 

including conference bridge 
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If conference center 
equipped for speech 
recognition, spoken 
commands are detected and 
sent to conference manager 
or conference server 
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Image data flows from originating conferee 
to conference server to other conferees; 
remaining separate from voice throughout 
their handling to and from the server 
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Data signals may represent dlsplayable 
images, visual cues to be displayed (e.g. bid 
to interrupt speaker), other cues, or service 
request changes to be Implemented by the 
conference server either directly or through 
the conference manager 
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Conference Master sends page data 
to PSTN conference server, before or 
during conference 
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During conference, conference server 
distributes pages one at a time in response 
to cues from conference master (data or 
vocal cues) 



If conferees have collaboration software, 
they may modify page currently displayed, 
upload it to server and cue server to 
distribute it to the others 
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Conferees send requests and/or other 
displayable data, server reacts to requests 
and distributes other data 
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Figure S 



55 

Participant sends request to server pertaining to J 



operation in PSTN voice signal path 



56 



if operation is permitted, server executes it or 
signals conference manager to do it 
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Server signals either acknowledgement of execution or 
"operation not permitted" indication to requestor 
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Figure 9 



Participant signals request to speak 
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Conference server illuminates icon associated 
with request in page currently being displayed 
to all conferees 
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Participant requests modification of voice 
signals sent to him/her 
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Conference server forwards corresponding 
signal to conference manager 
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Action implemented in voice path 
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Figure 11 




Gateway/Firewall 
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